Image super resolution (SR) is a method to obtain high quality images from low resolution input images. SR is widely applicable in video communication, object recognition, HDTV, image compression, among other situations where only a low resolution image is available. Generally speaking, low resolution images are generated by smoothing and down-sampling of target scenes by low-quality image sensors. The task of recovering the original high resolution (HR) image from single low resolution (LR) image is an inverse problem of this generation procedure. Ideally, the reconstruction error (or image likelihood term) should be minimized in the process.
Back-projection, an iterative process, has been used to efficiently minimize the reconstruction error. However, this process can lose significant amounts of information during the generation process. To overcome this difficulty, image prior terms have been used to regularize the inverse problem.
Two well-known image modeling priors are image smoothness prior and edge smoothness prior. Neighboring pixels are likely to have the same color, so various filtering/interpolation algorithms (for example, bilinear algorithm or bicubic interpolation algorithm) can be used to produce smooth high resolution images. Other smoothing techniques include minimizing the image derivative. For one dimensional case, a linear closed form solution can be used. However, the image smoothness prior is not valid at region boundary, such methods tend to produce over-smoothed results, thus reducing the image quality. To preserve edge sharpness, edge directed interpolation can be used to fit smooth sub-pixel edges to the image and to prevent cross-edge interpolation. However, locating high precision edge positions can be a non-trivial task.
When performing SR using the interpolation method, the chessboard effect that occurs needs to be removed. Given the low resolution input, high resolution edge position can be located by exploring the edge spatial smoothness prior, which means that smooth curves are generally preferred without other information. One technique reconstructs smooth approximation of all of the image level-set contours simultaneously to refine the edges and remove the chessboard effect. To avoid over-smoothness, hard constraints can be introduced, they are in essential information from the image likelihood.
Another technique considers all three color channels together, and infers the high resolution curves by multi-scale tensor voting. The HR images are recovered according to the extracted curveness map by a modified back-projection iteration. Yet another technique uses snake-based vectorization to achieve smooth boundary for icon image SR. Another image modeling prior technique for SR includes using two color image prior, which means that every pixel in a local neighbor-hood should be one of the two representative color, or a linear combination of them. The sparse derivative prior technique has also been used.
Instead of image prior modeling, the image exemplar can be used directly. The image is typically modeled as Markov Random Fields. Various candidates for each position are selected based on the low frequency information. Spatial consistency is enforced by pair-wise interaction, mainly on the overlapping region. The final discrete optimization problem is solved by belief propagation. This method can be applied to video sequence as well such as in domain-specific video SR. Two key issues usually need to be addressed for exemplar-based method: one is to find HR candidate patch efficiently, Locality Sensitive Hashing and KD-tree has been applied to speed up the searching. This method has also been applied to image primal sketches so that they only need to do the optimization on a chain structure. Yet other learning based methods have also been applied to infer the high frequency information from mid-frequency. For example, locally linear embedding can be used to learn the high dimension manifold.